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Executive Summary 



The goals of the International Academic Mobility (lAM) Initiative are to advance the 
development of international skills, knowledge and understanding among students and 
promote academic co-operation and institutional linkages among colleges and universities. 

The lAM Initiative includes the administration of two programs created by Human 
Resources Development Canada (HRDC) in 1995: the Canada- European Community 
Program for Co-operation in Higher Education and Training, and the Program for North 
American Mobility in Higher Education. These programs support consortium-based 
initiatives that are developed and carried out by Canadian universities and colleges in 
collaboration with institutions in other countries. 



Evaluation Process and Methodology 

The purpose of the formative evaluation was to assess the relevance, design and delivery, 
and the short-term benefits of the two lAM programs. 

The evaluation focused on projects accepted for funding between 1995 and 1998. During 
these years, lAM funded 30 projects under the Program for North American Mobility in 
Higher Education and 25 projects under the Canada-European Community Program for 
Co-operation in Higher Education and Training. 

The methodology used to conduct the formative evaluation included a literature review 
and surveys of Canadian project directors and educational partners, Canadian applicants, 
and participating Canadian students. 



Main Findings 

A summary of the main findings for the evaluation issues is presented below. The reader 
should note, however, that these findings should be viewed as being for the respondents 
to the surveys rather than for the larger population of lAM participants or unsuccessful 
applicants. 

Caution: Practical difficulties encountered in building the survey samples for the 
three surveys conducted for the formative evaluation resulted in small sample 
sizes. This limits inference to the population of lAM participants and biases the 
findings towards the few projects that participated in the surveys. 



a) Relevance 

The 40 surveyed faculty members believe that there is a continuing need for the 
lAM Initiative to promote student mobility. 
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The 29 surveyed unsuccessful applicants agreed that the lAM Initiative remains relevant 
and few of them (10 percent) found alternative funding to implement their projects after 
being denied lAM funding. 

Two-thirds of surveyed students indicated that they would have looked for other 
exchange opportunities had their lAM project not been available, but few (8 percent) 
believe that they would have had the opportunity to study abroad in the absence of the 
lAM project. 

b) Program Design and Delivery 
Reaching Institutions and Students 

The surveyed faculty members reported three primary sources for their knowledge of 
lAM: other faculty (43 percent), lAM promotional materials (28 percent), and university 
or college administration (15 percent). A slight majority of the surveyed students 
(54 percent) reported hearing about the lAM project from a faculty member, while others 
(16 percent) learned about lAM through recruitment posters or flyers, their academic 
department (17 percent), and from their international studies office (12 percent). 

The average stipend that surveyed students reported receiving was about $2,160 (with a 
range of 0 to $6,000). On average, the stipend covered about 77 percent of the 
incremental costs (identified by the students) of studying abroad. 

The lAM projects in the survey had an average of about $147,000 in lAM funding. 
The surveyed faculty members reported that, on average, lAM funding covered 
slightly less than three-quarters of their project’s cost. The surveyed faculty members 
also reported having to limit the number of students who could participate because of 
funding limitations. 

Less than half (45 percent) of the surveyed unsuccessful applicants said that they would 
not apply again to lAM. 

Design Elements 

The surveyed faculty members and unsuccessful applicants thought the clarity of the 
application process could be improved, particularly in the areas of selection criteria and 
administrative requirements. 

The surveyed faculty members criticised the lAM programs for funding only new 
projects. They wanted renewable funding for successful projects in order to build on 
those projects’ successes. Also, some of the surveyed faculty members wanted 
the duration of funding extended, citing the time needed to establish a 
functioning partnership. 
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Half of the surveyed faculty members reported being “very satisfied” or 
“somewhat satisfied” with their overall experience in administering a project under the 
lAM Initiative, while one quarter were “somewhat dissatisfied” or “very dissatisfied”. 

Overall, the surveyed faculty members were satisfied with their experiences collaborating 
with their partner institutions. 

About two-thirds of the surveyed students and faculty members indicated that orientation 
courses were offered to prepare students for the exchange. Eighty- seven percent of the 
surveyed students who received the orientation courses found them to be at least 
“somewhat helpful”, with almost one-third of the course recipients finding them 
“very helpful”. 

The surveyed students and faculty reported that the students experienced few problems 
with getting academic credit for their studies or work abroad. 

c) Short-Term Benefits 

Perceived Institution Effects 

The surveyed faculty members rated lAM’s contribution to establishing partnerships and 
enhancing existing partnerships as important to their institution. 

Formal collaboration occurred most often in the areas of academic credit recognition and 
transfer agreements, common curriculum materials, common courses or course modules, 
and faculty or staff exchanges. 

About two-thirds (68 percent) of the surveyed faculty members reported that the 
collaboration with their partner institutions continued after lAM funding ceased. 

Perceived Student Effects 

Overall, more than two-thirds (68 percent) of the surveyed students were “very satisfied” 
with their exchange experience. Only one percent was not satisfied. 

Most (75 percent) of the surveyed students agreed with the statement that their 
lAM experience helped them with their other academic programs. Also, almost half of 
the surveyed students believe that their academic progress abroad was greater than it 
would have been at their home institutions. 

Most of the surveyed student participants now employed full-time reported that their job 
is related to the studies or work they did in the lAM project (82 percent), and that they 
are using skills learned in the lAM project (88 percent). 

Non-mobile students became involved in the project in many ways, but primarily through 
taking courses developed with the partner institutions. Non-mobile students benefited 
from the international dimension of the jointly developed courses. 
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Conclusion 



The small sample sizes limit any inference from the survey results to the population of 
lAM participants and unsuccessful applicants as a whole. Some tentative conclusions are 
possible, however, based on the literature review and the information gathered by the 
formative evaluation: 

• The concept of international student exchange is widely supported within the academic 
community. Both faculty and students endorse the concept, as do studies of 
international exchange in other countries. 

• It is difficult to identify outcomes in terms of changes to career choice or career 
progress, but most students believe the experience was beneficial. 

• Consideration could be given to renewable funding for successful projects. The 
program will benefit from having experienced faculty run projects. To ensure access to 
a wider number of faculty members, the program could limit the number of awards 
within any five-year period. 

• Unsuccessful project applicants require additional feedback. These applicants should 
be viewed as a resource that with some coaching will become proponents of 
successful projects. 



IV 
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LLD Response to lAM Formative 

Evaluation Report 



May 1 , 2002 



Introduction 

This formative evaluation was undertaken to assess the relevanee, design and delivery, 
and short term impaets of the Program for North Ameriean Mobility in Higher Edueation, 
and the Canada-European Community Program for Co-operation in Higher Edueation 
and Training, whieh are administered under the International Aeademie Mobility 
Initiative. The evaluation proeess was initiated in 2000, five years after the programs 
started operation, and examined projeets that were approved between 1995-1998. 

The Eeaming and Eiteracy Directorate (EED) has reviewed the formative evaluation 
report that was recently completed. While sample sizes were not sufficient to be 
statistically reliable, nonetheless, we are still pleased with the overall findings and the 
insight that was provided on the lAM Initiative. 



Positive Findings 

Many of the findings throughout the report are positive. Eor example, participating 
faculty and students confirm that there is a continuing need to encourage international 
linkages and mobility activities within Canada’s post- secondary institutions, and that 
lAM continues to be relevant in providing funding and a framework for supporting these 
activities. Eaculty surveyed for the evaluation noted that lAM provides unique linkage 
and mobility opportunities that go beyond traditional bilateral exchange arrangements, 
and encourage innovation in international learning programs. 

The evaluation report indicates that projects are involving not only students who study 
abroad at partner institutions, but also a high number of students who do not study 
abroad. These non-mobile students benefit from the international dimension that the 
project brings to their education through, for example, taking courses for which the 
curriculum has been jointly developed with international partners. 

Eor the students who do study abroad, the financial stipend that they receive under 
lAM program is intended to assist with the incremental costs of studying abroad, above 
and beyond what the student would normally pay to pursue their education at their home 
institution. EED was pleased to note that the average stipend that surveyed students 
reported receiving covers 77% of the incremental costs that they identified. This is 
important, since 63% of surveyed students reported being recipients of some form of 
student loans. 
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Faculty reported considerable success in reaching agreements for credit transfer and 
recognition, and half of these faculties indicated that these agreements would benefit 
future students. Surveyed students reported little difficulty in receiving credit for their 
study or internships abroad. As well, most faculties reported that collaboration with their 
partners has continued since their lAM funding ended. 

Surveyed students reported improved language skills, knowledge of other cultures, and 
improved knowledge of international issues as a result of participating in lAM projects. 
Students also reported personal benefits including better ability to adapt to 
new situations. 



Areas for Improvement 

The lAM formative evaluation report also identified several areas where improvements 
could be made, particularly in the clarity of lAM’s application guidelines and selection 
processes. lAM guidelines, application procedures and selection processes are developed 
in co-operation with our international partner governments, and are reviewed annually to 
ensure clarity. While many improvements have been made since the early days of the 
initiative, we continue to refine the guidelines and selection processes based on 
comments received from our sponsors at Project Directors’ Meetings and from members 
of our External Review Committee. 

The report also identifies measures that would assist lAM in developing strong 
methodologies for a future summative evaluation of the program. These measures 
included continuous monitoring of students and faculty during and after project 
implementation, the use of a standard annual report template for project directors, and 
obtaining informed consent to contact students in the future for program 
evaluation purposes. 

lAM is already in the process of implementing many of these measures in response to 
internal program management review processes that occurred before or concurrently with 
the formative evaluation. These review processes included the Program Management 
Initiative (2000), the renewal of the Learning Initiatives Terms and Conditions under 
which the program operates (2000), and an audit that was conducted by HRDC’s Internal 
Audit Bureau (2001). 

A Results-Based Accountability Framework, outlining measurable performance measures 
for lAM’s programs, was developed as part of the renewal of the Terms and Conditions. 
As part of this process, lAM has developed student surveys and annual report templates 
that will permit continuous collection of comparable data on student experiences and 
project activities. Responses will be entered into a database that can be used in future 
evaluations. These tools are now in the process of being implemented and we will be in 
constant contact with our partners to monitor the extent to which they are user friendly 
and can be easily implemented. 
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lAM has drafted a participant information form that students will voluntarily complete to 
provide consent for HRDC to contact them in the future for the purpose of evaluating 
lAM’s programs. This form will be reviewed by HRDC’s privacy office, as well as our 
partnering institutions, to ensure that it complies with privacy laws both of the federal 
government and the post-secondary institutions that will be involved. Once finalised, this 
will assist in developing future survey frames. 



Conclusion 

At the first meeting of the lAM Steering Committee on March 23, 2001, a decision was 
made to revise the methodology that had been included in the Terms of Reference. Case 
studies and key informant interviews were replaced with a greater emphasis on the 
student survey, in an attempt to gather preliminary information on short-term benefits. 
While this change in methodology provided a series of “lessons learned” that will guide 
the development of future summative evaluation survey methodologies, LLD notes that it 
will also be important to include methodologies such as case studies and key informant 
interviews in future evaluations of lAM. This will be particularly important if 
establishing a statistically reliable sample of faculty and students continues to prove to 
be difficult. 
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1. Introduction 



The goals of the International Academic Mobility (lAM) Initiative are to advance the 
development of international skills, knowledge and understanding among students 
and promote academic co-operation and institutional linkages among colleges 
and universities. 

The lAM Initiative includes the administration of two programs created by Human 
Resources Development Canada (HRDC) in 1995: the Canada- European Community 
Program for Co-operation in Higher Education and Training, and the Program for 
North American Mobility in Higher Education. These programs support 
consortium-based initiatives that are developed and carried out by Canadian universities 
and colleges in collaboration with institutions in other countries. 

A formative evaluation was initiated in 2000 to assess the relevance, design and delivery, 
and short-term impacts of the two lAM programs. The formative evaluation also 
considered methodological issues, with particular emphasis on suggesting a design for the 
summative evaluations of the lAM programs. 

The formative evaluation focused on projects accepted for funding between 1995 and 
1998. During these years, lAM funded 30 projects under the Program for North 
American Mobility in Higher Education and 25 projects under the Canada-European 
Community Program for Co-operation in Higher Education and Training. To date, 
approximately 2,800 students have participated in the lAM programs, and about 1,500 of 
those students are estimated to have participated between 1995 and 1998. 

The methodology used to conduct the formative evaluation included a literature review, a 
survey of project directors and educational partners, a survey of unsuccessful Canadian 
applicants, and a survey of participating Canadian students. 

Caution: Practical difficulties encountered in building the survey samples for the 
three surveys conducted for this evaluation resulted in small sample sizes. The 
reader should note that the small sample size for each survey limits inference to 
the population of lAM participants, and biases the findings towards the few 
projects that participated in the surveys. Therefore, the reader must not draw 
inferences to the larger population of lAM participants or unsuccessful applicants 
based on the survey results presented in this report.^ 



1 .1 Overview of the lAM Programs 

Various consortia of Canadian universities and colleges, in collaboration with institutions 
in other countries, can apply for funding under the lAM programs to support their 
collaborative projects. A project typically consists of activities such as curriculum 



’ The report refers to the “surveyed students,” the “surveyed faculty memhers,” or the “surveyed unsuccessful 
applicants” to highlight the fact that these findings are based on very small samples, and the results describe only the 
survey respondents. 



Formative Evaluation of the International Academic Mobility Initiative 





development and the exchange process where Canadian students would study abroad. In 
some cases, students do not travel abroad (non-mobile students) but participate in other 
ways, such as through courses intended to increase their exposure to other cultures. A 
project director, usually a faculty member at a university or college, leads the project as 
part of the team of educational partners (faculty at other universities and colleges) that are 
included in the consortium for the project. 



The Program for North American Mobility in 
Higher Education 

HRDC and the Department of Foreign Affairs and International Trade announced the 
Program for North American Mobility in Higher Education (referred to in this report as 
Canada-NA Program) in June 1995. The Program is a partnership among the member 
governments of the North American Free Trade Agreement (NAFTA). It is a direct 
response to the commitment made in NAFTA to ""strengthen the special bonds of 
friendship and co-operation” between Canada, Mexico, and the United States. 

Since its inception, the Canada-NA Program has funded 50 projects (30 of which are part of 
this evaluation), in which approximately 1066 students have participated from 48 Canadian 
post-secondary institutions (this figure includes 33 universities, 1 1 community colleges, and 
4 university colleges).^ Five non-academic partners have also participated. 



The Canada-European Community Program for 
Co-operation in Higher Education and Training 

Announced at the end of 1995, the Canada-European Community Program for 
Co-operation in Higher Education and Training (referred to in this report as 
Canada-EC Program) also implements the objectives of a larger international agreement. 
In 1990, the Government of Canada and the member state governments of the European 
Community, now European Union (EU), adopted the Transatlantic Declaration. The 
Declaration was a response to the world events of 1989 and recognition that 
""transatlantic solidarity has played a historic role in preserving peace and freedom.” 
Among its many commitments, the Declaration included the strengthening of exchanges 
and joint projects, ""including academic and youth exchanges.” The Canada-EC Program 
is a direct result of that commitment. 

To date, the Canada-EC Program has funded 45 projects (25 are part of this evaluation) 
with approximately 1,725 students from over 56 Canadian academic institutions (this 
figure includes 42 universities, 11 community colleges, and 3 university colleges).^ 
Twenty-eight non-academic partners have also participated. 



^ The number of students is based on the number of students who will travel abroad as projected in the project 
proposals. Non-mobile students who remain at home but will participate through such activities as joint curricula 
development are not included. All numbers are for 1995 to the present. 

^ Please see footnote 2, above. 
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1.2 Evaluation Issues 



The formative evaluation of the two lAM programs was guided by the following 
eight evaluation issues. 

1 . Is there a continuing need to encourage exchange and co-operation among higher 
education institutions internationally? Is there still a need to prepare students for 
working in a global economy? 

2. What are the role and the nature of involvement of the I AM Initiative in encouraging 
exchanges and co-operation among higher education institutions from various 
countries? 

3. Would the participating institutions have been able to make progress in 
internationalizing their institution and/or strengthening student mobility if they had 
not had access to lAM funding? 

4. Has lAM encouraged international exchange and co-operation among Canadian 
higher education institutions (universities and colleges)? Specifically, in regard to: 

• setting up new multilateral partnerships between educational institutions and 
enhancing existing ones; 

• skills development and international academic work credit; 

• the development of common and/or shared curricula; and 

• the exchange of expertise and knowledge in new developments in higher education 
and training, including distance learning? 

5. Has lAM improved the quality of human resource development among mobile and 
non-mobile students at participating institutions? Specifically, in regard to: 

• increasing students’ knowledge of languages, cultures, and institutions; 

• providing affordable access to an “internationalized” curricula; 

• providing work placements/internships in a “foreign” milieu; and 

• partnerships beyond the educational field to include businesses, professional 
associations, public authorities, etc.? 

6. Are the partnerships created through the I AM program continuing beyond the 
program funding? What mechanisms are being used to promote sustainability? 

7. Has lAM reached targeted institutions’ faculties and students? 

8. Are there design elements that constitute obstacles to effective implementation? If so, 
can and how could they be changed? 
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1 .3 Overview of the Report 

This report presents the main findings of the formative evaluation and includes the 
following: 

• An overview of the methodology used to conduct the formative evaluation; 

• A profile of the respondents to the surveys and their lAM projects; 

• A summary of the findings for each evaluation issue; 

• Recommendations for the summative evaluation of the lAM Initiative, based upon the 
experiences of this evaluation; and 

• A brief overview of the main conclusions. 
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2. Methodology 



This section provides highlights of the multiple lines of evidence developed for the 
formative evaluation, some of the challenges encountered, and profile information that 
presents a picture of the surveys’ respondents. 



2.1 Multiple Lines of Evidence 

The formative evaluation obtained its findings from multiple lines of evidence. 
A summary of each methodology is presented below. Readers interested in more detail 
can request the evaluation’s technical reports from the Evaluation and Data Development 
Branch of HRDC. 



Literature Review 

The literature review focused on reviewing literature on the perceived benefits of 
academic mobility on students and academic institutions. The available literature was 
compiled from an Internet scan, on-line document retrieval services, web-based 
bibliographies, databases for periodicals, and several library catalogues 

(public and university). 



Survey of Canadian Project Directors and Educationai 
Partners 

The evaluation included a mail survey with 125 project directors and educational partners 
identified through the lAM database. The overall response rate for the survey of project 
directors and educational partners was 32 percent (40 of 125). The group response rates 
were 52 percent for project directors (25 of 48) and 19 percent for educational partners 
(15 of 77).^ 

Survey of Unsuccessfui Canadian Appiicants 

The evaluation included a fax survey with 114 unsuccessful applicants identified in 
lAM administrative files. Of the 114 identified unsuccessful applicants, 29 responded to 
the survey for a response rate of 25 percent. 



Because project directors had more direct contact with students, they will typically have more detailed knowledge 
about all aspects of the project and student participation. Also, since the responses between the two groups did not 
yield statistically significant differences, they are combined in the report of the survey’s findings. 



Formative Evaluation of the International Academic Mobility Initiative 




Survey of Participating Canadian Students 

The evaluation included a telephone survey of 113 students who registered by telephone, 
mail-in card, or the Internet. Non-mobile students were not targeted by this survey. 

The following strategy was implemented to build the sample for the survey of 
participating Canadian students: 1) Project directors and educational partners received 
letters from HRDC requesting that they send names of students participating between 
1995 and 1998 to the registrar of the institution; 2) Simultaneously, registrars received 
letters identifying the faculty who would be sending the names plus a request that 
packages prepared for the purpose of the evaluation be sent to the student's permanent 
address of record (typically a parent). These packages contained a letter to the parent 
requesting their assistance in locating the student by forwarding a letter; and, 3) Students 
received letters from their parents inviting them to register for a survey by phone 
(toll free line), mail, or through a web site. 

Of the estimated 1,500 students who participated in lAM between 1995 and 1998, 
161 registered for the survey and 113 completed interviews (after multiple attempts to 
make contact). 



2.2 Evaluation Challenges 

Given the amount of time that has passed since the programs were initiated in 1995/1996, 
a decision was made at the lAM Evaluation Steering Committee meeting to include some 
of the elements of a summative evaluation in the formative evaluation. Specifically it was 
decided to attempt to measure early impacts, and it was hoped that lessons drawn from 
the consideration of early impacts would inform the future summative evaluations on 
how to measure longer-term impacts. 

A number of difficulties were encountered during the course of the formative evaluation. 
In the case of the literature review: 

• There were very few evaluations of the incremental benefits of academic mobility 
programs. Those that exist tend to focus on the short-term effects on students and on 
students’ satisfaction with their exchange experience. In addition, the scope of the 
literature is restricted to studies of student mobility in the university setting. 

• The studies had methodological limitations. First, most studies rely on self-reported 
data, with no external check to determine the effects of academic mobility on students. 
Second, none of the studies use a comparison group of non-exchange students. 
Whether the effects reported by exchange students are experienced to a similar degree 
in non-exchange students remains unanswered. 

In the case of the three surveys: 

• Three years had elapsed between the conclusion of some projects and the start of the 
evaluation. The lAM database was not systematically updated after project conclusion 
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and the contact information for some project directors and educational partners and 
unsuccessful applicants was invalid. 

• While most project directors could supply names of the participating students, 
information on the current or parental address of the students was unavailable. 

• Confidentiality policies at many universities and colleges required an informed consent 
process. This requirement was coordinated through an information flow from faculty 
members to registrars to the parents of students and finally to the students. 

• The amount of time already devoted to the administration of the project by the 
preparation of the annual report and participation in the audit process were indicated as 
reasons for not participating in the surveys. 

• Faculty may have little incentive to spend time on surveys since they become involved 
in lAM programs from an altruistic motive to offer students the opportunity to study 
abroad. The lAM Initiative does not directly fund academic research or publication. 

Caution: The survey results presented in this report involve small sample sizes 
for each of the surveys. This limits inference to the population of 
I AM participants and biases the findings towards the few projects that 
participated in the surveys. Therefore, the reader must not draw inferences to the 
larger population of lAM participants or unsuccessful applicants based on the 
survey results presented in this report. 



2.3 Profiles 

This section examines profile information to provide a picture of the respondents to the 
surveys and their lAM projects. 



Profile of the Projects 

Faculty from 33 of the 55 lAM projects participated in the survey. Their responses 

provide the following overview of project characteristics: 

• The projects covered a variety of disciplines, with agricultural and natural resources 
(18 percent), biological sciences (15 percent), and five other areas (business 
management, education, engineering and applied sciences, social sciences, and health 
professions and occupations at 12 percent each) heading the list. 

• The foreign academic partners are dominated by the United States and Mexico, 
the only two foreign partners in the Canada-NA Program. Partners in the 
Canada-EC Program are from 14 of the 15 member states of the EU, with Erance and 
Sweden having the most academic partners. Eew of the projects in the survey had 
non-academic partners. Table 1 provides the number of academic and non-academic 
partners by country for the 33 projects. 
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# of academic 
partners 


# of non-academic 
partners 


# of students 


Canada 


64 


10 




Mexico 


26 


2 


109 


United States 


24 


2 


88 


France 


6 


3 


57 


Sweden 


6 


- 


43 


United Kingdom 


5 


- 


29 


Netheriands 


5 


- 


26 


Germany 


3 


- 


24 


Spain 


4 


- 


19 


Beigium 


4 


- 


14 


Ireiand 


2 


- 


11 


Denmark 


1 


- 


7 


Portugai 


2 


- 


5 


Finiand 


3 


- 


4 


Austria 


1 


1 


2 


Italy 


2 


- 


2 


Greece 


1 


- 




Other 


2 


1 




Total 


161 


19 




Source: Survey of project directors and educational partners (n=40) 



• Students could participate in projects as mobile students, who go abroad to a “host 
institution” for study, research or work. The total number of mobile students travelling 
to each country is shown in Table 1. Alternatively, students could participate in 
projects as non-mobile students, who participate in the project while remaining at their 
Canadian institution (the “home institution”). Based on the faculty survey, a median of 
1 1 mobile and 5 non-mobile students participated in the project from each institution. 
The distribution of mobile and non-mobile students participating in a project from each 
institution is provided in Table 2. Most institutions had fewer than 10 students 
participating in a project; 3 percent of the surveyed faculty members reported that their 
institution had over 41 mobile students in an lAM project. 



Table 2 

Number of mobiie students and non-mobiie students reported by facuity members (n=40) 




Mobiie students 


Non-mobiie students 




# of facuity 
reporting 


% 


# of facuity 
reporting 


% 


0-10 students 


18 


45% 


16 


40% 


11-20 students 


12 


30% 


4 


10% 


21-40 students 


6 


15% 


3 


8% 


41-60 students 


1 


3% 


2 


5% 


61 or more 


- 


0% 


2 


5% 


Don’t know 


3 


8% 


13 


33% 


Source: Survey of project directors and educational partners 
Note: Totals may not sum to 100 percent due to rounding. 
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• The average stay abroad for mobile students was 20.2 weeks, with a median of 
9 weeks, showing that a few extended stays occurred (a maximum of 108 weeks in 
one case). Sixty percent (n=24) of faculty reported student sojourns of 18 weeks 
or less. 

• Half (53 percent or n=21) of the surveyed faculty members reported having students in 
internships (which includes employment or fieldwork) while abroad. The average 
internship experience reported was 10.6 weeks. 



Profile of Surveyed Project Directors and 
Educational Partners 

On average, the surveyed faculty members (n=40) have taught at the post- secondary level 
for 21.4 years and at their current institutions for 17.6 years. A slight majority (53 percent 
or n=21) are full professors and almost one-fifth (15 percent or n=6) are associate 
professors. None were assistant professors. 

The project directors and educational partners appear to receive substantial external 
funding, with an average of 52 percent of their current research and/or teaching funded 
by agencies outside of their post- secondary institution. 

Three-quarters (78 percent or n=31) of the surveyed faculty members were engaged in 
collaborative research with foreign institutions during 1998-2001. Overall, just over a 
quarter (28 percent) of faculty members' research involved international collaboration. 



Profile of the Surveyed Unsuccessful Applicants 

In the case of respondents to the survey of unsuccessful applicants (n=29), 
three (10 percent) applied to the Canada-NA Program, twenty (69 percent) applied to the 
Canada- EC Program, and six (21 percent) applied to both programs between 1995 and 
1998. Most applicants applied once (48 percent or n=14) or twice (38 percent or n=ll) to 
one of the lAM programs during this time period. Few applied more than twice 
(10 percent or n=3). Four respondents withdrew an application during 1995-98. 
Five respondents reported having at least one successful application to lAM during 
1995-98; however, none of these respondents appeared in the lAM database of project 
directors and educational partners, nor were they identified participants during the 
evaluation’s contacts with other participating faculty members. 



Profile of Surveyed Participating Students 

The completed interviews with participating Canadian students (n=113) provides the 
following profile information for the survey’s respondents. 
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Academic Profile 



The interviewed students came from 28 Canadian post-secondary institutions. 

Ninety-four percent (n=106) of the surveyed students participated in an lAM project in 
the same area as their major or area of concentration. While the surveyed students had a 
variety of academic majors or areas of concentration, the areas with the greatest 
concentration of survey respondents are in law (17 percent or n=19), engineering and 
applied science (13 percent or n=15), and agriculture or natural resources (12 percent or 
n=14). 

Two-thirds (67 percent or n=76) of the surveyed students were undergraduates at the time 
of their participation in an lAM project. Three-quarters (76 percent or n=86) were beyond 
the second year of their degree program. 

Almost half of the respondents (47 percent or n=53) reported an average grade of A- or 
better, and another third (32 percent or n=36) reported a B-i- average at the time they 
applied for the lAM project. Few had an average grade of less than B (2 percent or n=2). 

Participation Profile 

All of the surveyed students went abroad as part of their lAM project: 70 percent (n=79) 
studied abroad; 21 percent (n=24) studied and did an internship abroad; 5 percent (n=6) 
worked or did an internship abroad; 4 percent (n=4) did not take formal courses but went 
abroad for other academic activities (e.g., for a conference, to do research). 

Surveyed students reported a median stay abroad of 16 weeks, with a range of one week 
to 80 weeks. 

Sixty-four (57 percent) of the surveyed students were part of the Canada-NA Program 
and travelled to either the United States or Mexico. Forty-nine (43 percent) of the 
surveyed students participated in the Canada-EC Program, and over half (n=27 of 49) of 
them travelled to either Sweden or the United Kingdom. 

Personal Profile 

Almost two-thirds of the surveyed students are women (64 percent or n=72). 

Over half (53 percent or n=60) of the surveyed students are now working full-time, while 
a little more than a third (35 percent or n=39) are studying full-time. 

The surveyed students who are employed full-time hold jobs in a variety of areas. The 
types of employment most reported by the students are: health or medicine (27 percent or 
n=16 of 60), sales or service industries (12 percent or n=7), and government (10 percent 
or n=6). 

The annual income levels of the surveyed students who are currently working full-time 
(n=60) are provided in Table 3. No students working full-time reported making less than 
$20,000 per year. 



Formative Evaluation of the International Academic Mobility Initiative 








# 


% 


$20,001 to $35,000 


14 


23% 


$35,001 to $50,000 


24 


40% 


$50,001 to $75,000 


17 


28% 


$75,001 to $100,000 


1 


2% 


Don’t know 


4 


7% 


Source: Survey of participating students 



More than 70 percent of the surveyed students reported that their parents made less than 
$75,000 a year. During their education, over two-thirds (67 percent or n=76) of the 
surveyed students received financial assistance from their parents. A similar percentage 
(63 percent or n=71) used student loans to help cover their educational costs. 
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3. Main Findings 



This section presents the main findings obtained from multiple lines of evidence used 
to examine the evaluation issues for relevance, design and delivery, and short-term 
benefits. Given the small sample sizes, the results should be viewed as being for the 
respondents to the surveys rather than for the larger population of lAM participants or 
unsuccessful applicants. 



3.1 Relevance 

The formative evaluation examined three evaluation issues regarding relevance: 

• Is there a continuing need to encourage exchange and co-operation among higher 
education institutions internationally? Is there still a need to prepare students for 
working in a global economy? 

• What is the role and the nature of involvement of the lAM Initiative in encouraging 
exchanges and co-operation among higher education institutions from various countries? 

• Would the participating institutions have been able to make progress in 
internationalizing their institution and/or strengthening student mobility if they had not 
had access to lAM funding? 



Findings on Reievance 

Although many Canadian post-secondary institutions have international exchange 
agreements, less than one percent of Canadian university students studied abroad 
during the 1997/98 academic year. 

The surveyed faculty members believe that there is a continuing need for the 
lAM Initiative to promote student mobility. They cite the importance of 
lAM funding and that lAM provides unique opportunities, for example, to deepen 
exchange relationships and bring together a wide range of students. 



Eighty-eight percent of the surveyed faculty members reported that their institution or 
department participates in other non-IAM funded exchanges. Fifty-six percent (n=22) of 
the surveyed faculty members reported an increase in participation in non-IAM funded 
exchange programs in the last five years. 

A recent study by the Association of Universities and Colleges of Canada (AUCC) also 
shows that institutional interest in providing exchange opportunities is high, with 
90 percent of responding institutions indicating a medium or high interest in providing 
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student exchange experiences.^ In 1999, Canadian universities were participating in 
almost 2,500 international exchange agreements. These agreements are providing 
exchange experiences for several thousand students a year. For example, during the 
1997/98 academic year, institutions reported 5,058 Canadian students studying abroad as 
part of an exchange.^ However, even this level of participation means that less than one 
percent of Canadian university students (full-time graduate and undergraduate students) 
were studying abroad in an exchange program during the 1997/98 academic year. 

The AUCC study reports a need for continued funding of student mobility programs. This 
position is reinforced by a focus group conducted by HRDC/IAM with project directors 
in the Canada-NA Program. The focus group study indicates that project directors 
believed that even with increasing participation in international activities by their 
universities/colleges, the Canada-NA Program is still needed to promote student mobility. 
The project directors noted that funding for international mobility continues to be scarce, 
and that most institutions still have room to improve their international programs. 

Although the majority of the faculty members surveyed for the lAM evaluation are in 
institutions where exchange participation is on the rise, 77 percent of those faculty 
believe that lAM is still needed to support international exchange at their institution. 
When asked to explain why they believe that the lAM Initiative is still needed, the 
surveyed faculty members focused primarily on the importance of lAM funding but also 
noted that lAM provides unique opportunities, such as allowing institutions to enhance 
their programs and expertise through deepened relationships and bringing a wide range of 
students together. 



The surveyed unsuccessful applicants also agree that the lAM Initiative remains 
relevant. Their view is supported by the experience that few of them found 
alternative funding to implement their projects after being denied lAM funding. 



The respondents to the survey of unsuccessful applicants also noted the need for more 
funding for international exchanges. They unanimously agreed (n=29) that a continuing 
need exists to encourage international exchange programs and co-operation among higher 
education institutions internationally. They hold this view even though they reported that 
their departments and institutions already participate in student exchanges and that their 
own participation in international exchange programs has increased in the last five years. 

The continued need for the funding of exchange projects is illustrated by the fact that 
over four-fifths (86 percent or n=25) of the surveyed unsuccessful applicants reported 
that their project did not proceed after being denied lAM funding. Only three of the 



^ The following is based on information in Jane Knight, Progress and Promise: The AUCC Report on 
Internationalization at Canadian Universities (Ottawa: Association of Universities and Colleges of Canada, 2000). 

^ Knight notes that institutions are not systematically keeping this data, so this is an estimate. 

’’ In March 2000, a focus group was held with 13 project directors in Austin, Texas during the annual project 
director’s meeting for the Program for North American Mobility for Higher Education. The focus group comments 
(as summarized by the moderator) were used for developing the main evaluation issues of the current evaluation of 
the Canada-NA Program and the Canada-EC Program. 
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respondents (10 percent) found alternative funding for their projects, and all of them had 

o 

to modify the projects due to lack of funds. 

The survey of unsuccessful applicants also appears to show a greater need for 
opportunities in more innovative areas of international exchange, which are areas 
supported by the lAM Initiative. Unsuccessful applicants reported most faculty 
involvement in the more traditional forms of international exchange (i.e., student 
exchanges, research with foreign partners, faculty exchange), while less traditional areas 
of international exchange had less faculty involvement. About a quarter of unsuccessful 
applicants reported participating in the development of shared or common curricula with 
foreign educational institutions or the development of joint degree programs during the 
past two years. 

The surveyed students indicated a substantial interest in student exchange, and 
two-thirds said that they would have looked for other exchange opportunities had 
their lAM project not been available. At the same time, however, the students 
appear to view exchange opportunities as limited and very few of them believe 
that they would have had the opportunity to study abroad in the absence of their 
lAM project. 



The questionnaire for participating students considered lAM’s relevance by asking about 
the students’ desire for exchange opportunities and what they would have done had the 
lAM exchange not been available. Over two-thirds (67 percent of n=76) of the surveyed 
students said that they would have looked for other student exchanges had the 
lAM project not been available. The continued need for lAM may be seen in the fact that 
only 8 percent (n=9) think that they would have studied abroad in the absence of their 
lAM project. 



3.2 Program Design and Delivery 

The formative evaluation examined two issues regarding design and delivery: 

• Has lAM reached targeted institutions’ faculties and students? 

• Are there design elements that constitute obstacles to effective implementation? If so, 
can and how could they be changed? 



Reaching Institutions and Students 

The surveyed faculty members were from many different institutions, and the 
projects were in a variety of academic disciplines. The surveyed students were 
also from many institutions and academic fields of interest. 



Three percent (n=l) of unsuccessful applicants did not respond to this question. 
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The project profile information presented in Section 2 indicated that the lAM projects 
were in a variety of disciplines including agricultural and natural resources, biological 
sciences, business management, education, engineering and applied sciences, social 
sciences, and health professions and occupations. 

Also, Section 2 indicated that the surveyed students came from 28 post-secondary 
institutions. Their areas of study included law, engineering, applied science, agriculture 
and natural resources. 

The surveyed faculty members reported three primary sources for their 
knowledge of lAM: other faculty, lAM promotional materials, and university or 
college administration. 

The surveyed faculty members reported having to limit the number of students 
who could participate because of funding limitations. Some faculty also reported 
difficulties with recruitment due to the stipend amount. 

Overall, surveyed faculty reported three primary sources for their knowledge of lAM: 
other faculty (43 percent or n=17), lAM promotional materials (28 percent or n=ll), and 
university or college administration (15 percent or n=6). These results show that while 
investment in lAM promotional materials is well spent, there is also a strong “word of 
mouth” quality to how faculty members hear about the lAM Initiative (58 percent of 
faculty surveyed hear of lAM through other faculty and/or administrators). 

Faculty recruited students mainly by advertising the program with posters/flyers 
(83 percent or n=33), personally approaching students (63 percent or n=25), announcing 
the project in class (73 percent or n=29), and asking other faculty to assist with recruiting 
(55 percent or n=22). A slight majority of the surveyed students (54 percent or n=61) 
reported hearing about the I AM project from a faculty member, while 16 percent (n=18) 
learned of lAM through recruitment posters or flyers. The surveyed students also learned 
about the project from their academic department (17 percent of n=19) and from their 
university or college international studies office (12 percent or n=14). 

Less than half (40 percent or n=16) of the surveyed faculty members reported absolutely 
no obstacles in recruiting students, meaning that most had experienced some difficulties. 
These obstacles took many forms, including finding students available for an exchange 
(28 percent or n=ll), the stipend amount (23 percent or n=9), difficulty finding qualified 
students (13 percent or n=5), home institutional structures, such as academic programs or 
curricula that made scheduling exchanges difficult (13 percent or n=5), and insufficient 
language skills (10 percent or n=4). In spite of the obstacles in recruiting, however, 
almost half (48 percent or n=19) of the faculty had to limit the number of mobile students 
accepted for the project. 



The surveyed students estimated that the cost of studying abroad through the 
lAM program was, on average, about $2,780 more that the cost of studying at the 
home institution for the same period of time. The average stipend for the surveyed 
students was about $2,160 (with a range of 0 to $6,000). Therefore, on average, the 
stipend covered about 77 percent of the incremental costs (identified by the 
students) of participating in lAM. 
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The evaluation conducted by the European Union of its Co-operation Programme in 
Higher Education and Training between the European Community and Canada 
(the EU’s title for its side of the Canada-EC Program) found that a significant proportion 
of the costs of studying abroad are borne by the individual student and family. 

The surveys of participating students and faculty conducted for the lAM evaluation also 
found that the student pays much of the cost of studying abroad.^ 

• Almost two-thirds (64 percent or n=72) of surveyed students reported that the cost of 
the lAM project was higher than the cost of studying at their home institution for the 
same period of time. Surveyed students estimated that on average, the incremental cost 
of participating in lAM was $2,778 (median of $2,000). 

• The surveyed students reported an average stipend of $2,162. The stipend amount 
ranged from zero (8 percent or n=9) to $6,000 (1 percent or n=l). These figures appear 
to be in agreement with the amounts reported by surveyed faculty: an average stipend 
of $2,870, with the amounts ranging from $1,000 to $5,400. 

• The stipends received by the surveyed students did not cover the full cost of 
participation in the exchange. Students reported an average cost (excluding 
reimbursements and travel costs) of $4,798 (median cost of $3,500), and 82 percent 
(n=85) stated that the stipend did not completely cover their costs associated with the 
lAM project. 

• A minority of students (34 percent or n=38) said they found additional funding to 
offset the costs of the exchange. These surveyed students averaged $4,654 in outside 
funding, with a median amount of $2,500. Most reported outside funding from 
bursaries or scholarships (24 percent or n=9 or 38), student loans (16 percent or n=6), 
university or college funds (13 percent or n=5), and parents/family (8 percent or n=3). 

• Eaculty used very different methods for determining the amount of funding to be 
awarded to students. The three most common methods are: basing the decision on the 
cost of the project to the student (38 percent or n=15); giving each student identical 
amounts (20 percent or n=8); and determining the amount based on the projects budget 
(15 percent or n=6). 



The surveyed faculty members reported that lAM funding covered slightly less 
than three quarters of their project’s costs. 



The projects in the survey had an average of $147,283 in funding. This amount covered, 
on average, 73 percent of the total project cost. Table 4 shows the allocation of the 



* It should be noted that for this evaluation, the evaluator cannot comment on the relative wealth of student 
participants or their parents due to the small sample size. 

The survey asked students to provide an estimate of the overall cost to them of the exchange project. Students were 
told to exclude any amounts for which they were reimbursed (i.e. the stipend) or that were spent on pleasure travel. 
Students were then asked if this was more than they would have spent at their home institution and, if so, how much 
more. 

** Base for question is the students who received a stipend. 
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project funding reported by the surveyed faculty members. Many of the survey 
respondents did not know or did not respond to this question, however. 







Actual expenditures 
[Mean percentage] 


Student mobility and related costs 


60% 


Faculty-staff travel 


21% 


Administrative support 


10% 


Curriculum development and distance learning 


6% 


Promotional materials 


1% 


Costs of hosting partner meetings 


4% 


Other costs 


8% 


Source: Survey of project directors and educational partners 
Note: Means do not include "don’t know" and "no response" answers. 
Base: Projects of the surveyed faculty members. 



Slightly over half (55 percent or 18/33) of the projects in the survey received funding 
from lAM only. Thirteen projects reported additional sources of funding, which were 
mainly from other partner institutions or the general budget of their post- secondary 
institution.^^ 



Almost half of the surveyed unsuccessful applicants said that they would not 
apply again to lAM. All of them considered the application process too time 
consuming, and many either did not receive feedback on why their application 
was rejected or did not consider the reasons given to he appropriate. 



When the unsuccessful applicants were asked if they would apply to the lAM Initiative in 
the future, a little over one-third (35 percent or n=10) responded that they would 
definitely apply again. Forty-five percent (n=13) reported that they would not apply again 
and they gave the following reasons: 

• The application process is too time consuming (n=13, which corresponds to all of those 
who said that they would not apply again); 

• The decision process seems unfair and arbitrary (n=8 of 13); 

• The consortium partner requirements are too difficult to comply with (n=7 of 13); 

• Funding for projects is insufficient (n=5 of 13); and 

• The requirement for academic partner contributions is a burden (n=3 of 13). 

Also, many of the surveyed unsuccessful applicants found feedback on their applications 
lacking. Over half (52 percent or n=15) of the unsuccessful applicants reported not 
receiving reasons for the rejection of their application. Among the group that did receive 



Circa Group, The Evaluation of the Co-operation Programme in Higher Education and Training Between the 
European Community and Canada (contracted by E.U. Directorate General XXII, 1999). 
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reasons for their rejection (45 percent or n=13), over three-fifths (62 percent or n=8 of 
13) did not consider the reasons for their application’s rejection appropriate. 



Design Elements 

The surveyed faculty members and unsuccessful applicants thought the clarity of 
the application process could be improved, particularly in the areas of selection 
criteria and administrative requirements. 



Both the surveyed faculty members and unsuccessful applicants were asked to comment 
on the application process. The results of the two surveys were very similar. Both groups 
found the clarity of the lAM guidelines and forms to rate less than 4.0 (on a scale of 1 to 
5, where 1 means "very unclear" and 5 means "very clear"). They considered eligibility 
requirements and the types of costs covered to be the most clearly presented elements of 
the application forms and guidelines. The lowest rated items by the faculty members were 
selection criteria (3.2) and administrative requirements (3.1). For the unsuccessful 
applicants, it was activities eligible for support (3.4) and selection criteria (2.9). 

The surveyed faculty members criticized the lAM programs for funding only new 
projects. They wanted renewable funding for successful projects in order to build 
on those projects’ successes. 

Also, the surveyed faculty members were neutral about the level and the duration 
of project funding. 



When asked about their level of agreement with the program elements of lAM (on a scale 
of 1 to 5, where 1 means “strongly disagree” and 5 means “strongly agree”), the surveyed 
faculty members evidenced mild agreement, with no program element rating a 4.0 or 
above. Faculty members: 

• Had the most positive reaction to the requirement of a national distribution of partners 
(mean of 3.6); 

• Were essentially neutral on the level and duration of project funding, with current 
funding levels receiving a 3.3, and funding duration and the required institutional 
contribution receiving 3.0 ratings; and 

• Disagreed with the requirement that funding was only available for new projects 
(i.e., no funding to continue a pre-existing project), with this element receiving a 
2.0 rating and indicating that many faculty members believed that successful projects 
should be eligible for renewed funding. 



The only item rating 4.0 or more was the eligibility requirements (by participating faculty only). 
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Half of the surveyed faculty members reported being “very satisfied” or 
“somewhat satisfied” with their experience in administering a project under the 
lAM Initiative. Fifteen percent of the surveyed faculty members characterized 
their experience as neutral, while 25 percent were “somewhat” or 
“very dissatisfied.” 



The surveyed faculty members were divided in assessing their overall experience in 
administering a project under the lAM Initiative. Half (53 percent or n=21) reported 
being “very satisfied” or “somewhat satisfied”, 15 percent (n=6) characterized their 
experience as neutral, while 25 percent (n=10) were “somewhat dissatisfied” or 
“very dissatisfied.” 



Overall, the surveyed faculty members were satisfied with their experiences 
collaborating with their partner institutions. 

The most common forms of co-operation reported by the faculty members were in 
the areas of credit recognition and transfer agreements, common curricular 
materials, and faculty or staff exchanges. 

Most consortium collaboration consisted of informal agreements. The main 
exception was collaboration in the area of credit recognition and transfer 
agreements (where half of the agreements were formal). 

To improve co-operation among partners, faculty listed the need for more contact 
with their partners and more money for travel and other consortium-building 
expenses. 



Overall, the surveyed faculty members expressed satisfaction with their experiences of 
collaborating with partner institutions. Almost half (48 percent or n=19) of them reported 
being “very satisfied” and over one-third (38 percent or n=15) were 
“somewhat satisfied.” Four faculty members (10 percent) felt “somewhat dissatisfied” 
with the experience, and none rated their level of satisfaction as “very dissatisfied.”^'* 

The questionnaire of participating faculty members asked them to discuss the forms of 
collaboration their consortium engaged in. The key findings are: 

• Collaboration (either formal or informal agreements) occurred most often in the 
following areas: credit recognition and transfer agreements (65 percent), common 
curricular materials (58 percent), and faculty or staff exchanges (56 percent). 

• Collaboration with consortium partners tended to be through informal rather than 
formal agreements. The main exception was in the area or credit recognition and 
transfer agreements, where half of the agreements were formal. Consortia used 
informal agreements for all other areas of collaboration, if they collaborated at all. 



One faculty member was neutral, and one did not respond to the question. 



Formative Evaluation of the International Academic Mobility Initiative 






• About 41 percent of the surveyed faculty members reported that no collaboration or 
discussion surrounding collaboration occurred in the areas of partnerships with 
non-academic organizations. 

• Even though the program was not aimed at developing the collaboration in the areas of 
joint degree programs and common certification programs, more than two-thirds of the 
faculty members reported that it occurred during the project. 

Generally, the surveyed faculty members were satisfied with all forms of partner 
co-operation included in the survey. On average, as shown in Table 5, the faculty 
members rated their satisfaction with various areas at 3.4 or above (using a scale of 1 to 
5, where 1 means “very unsatisfied” and 5 means “very satisfied”). The surveyed faculty 
members reported the most satisfaction in facilitating entry of students into the country 
and providing a quality experience for students. A close second was co-operation 
concerned with issues of academic credit (i.e. procedures for evaluating student work, 
academic credit transfers, schedules for exchanges). 

Table 5 shows that ranking among the lowest satisfaction were the following: 

• Co-operation in preparing students academically, culturally, and linguistically for the 
exchange ranked among the lowest in satisfaction, as did finding qualified students. 

• Securing internships for students had the lowest satisfaction rating. This is an area 
where three-fourths of the faculty reported that they relied on the host institution for 
this assistance. 

• Although developing course content and curricula were two of the most common forms 
of collaboration, the co-operation in these areas received relatively low 
satisfaction ratings. 
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Mean 

(1=very unsatisfied 
5=Very satisfied) 


N/A 


Facilitating entry of students into country 


4.4 


3% 


Providing a quality experience for students 


4.4 


0% 


Establishing procedures for evaluating student work 


4.2 


17% 


Negotiating academic credit transfers 


4.1 


11% 


Negotiating schedules for exchanges 


4.1 


11% 


Involving non-mobile students 


4.0 


50% 


Timely communication about consortium issues 


4.0 


0% 


Sharing resources, expertise, technology 


4.0 


6% 


Sufficient academic preparation for students 


3.9 


0% 


Recruiting qualified students 


3.8 


3% 


Sufficient cultural preparation for students 


3.8 


6% 


Sufficient language preparation for students 


3.6 


19% 


Developing course content and curricula 


3.6 


25% 


Securing internships for students 


3.4 


47% 


Source: Survey of project directors and educational partners 

Note: Mean does not include "don't know," "no response," and "not applicable" answers. 



Faculty cited many ways to improve co-operation among consortium partners: 

• more contact with partner (n=8); 

• more money for travel or other consortium-building expenses (n=4); 

• better management of projects (e.g., have two project directors, create more trust) 
(n=3); 

• choosing partners with experience in co-operative projects (n=2); and 

• more time to develop projects (n=2). 

About two-thirds of the surveyed students and faculty members indicated that 
orientation courses were offered to prepare students for the exchange. Most of the 
students who received the courses found them helpful. 



Sixty percent (n=24) of the surveyed faculty members reported that their projects 
provided students with orientation courses to prepare the students for the culture, politics, 
business climate, or educational system of the host country. Similarly, almost two-thirds 
(63 percent or n=71) of surveyed students reported that their home institutions provided 
orientation programs or meetings to prepare them for their lAM project. 
Eighty-seven percent (n=62 of 71) of these students found the orientation offerings at 
least “somewhat helpful”. 
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The one-third (35 percent or n=40) of surveyed students who did not receive any 
orientation did not appear to have greatly missed it. A majority (55 percent or n=22 of 
40) of them reported that they did not wish they had received any orientation. The results 
for the host institution were, for the most part, similar. 

The types of contacts that students had with host institutions prior to going abroad took 
many forms, with most of the contacts being either written or e-mail contact rather than 
in-person contact. For over four-fifths (84 percent or n=95) of the students surveyed, their 
home institution assisted them in contacting their host institution prior to going abroad. 
While over one-third (39 percent or n=44) found the amount of prior contact “completely 
adequate,” almost half (48 percent or n=54) felt that they “could have used more” 
contact, and 12 percent (n=14) of surveyed students considered their prior contact 
“totally inadequate.” 



Most of the surveyed students reported being satisfied with the assistance they 
received from their home and host institutions for their exchange. 



The surveyed students were asked about the support they received from both their home 
and their host institutions in a range of areas. As shown in Table 6 (which provides the 
survey data for home institution supports): 

• Most types of assistance received by the majority of surveyed students came from the 
home institution. 

• Students generally reported that they received the supports they considered to be the 
most important. In most cases, the level of support received and the importance 
students placed on the support seem to track each other. The two main exceptions were 
registration and course selection for the host institution and arranging living 
accommodations. Students ranked these two types of assistance as being very 
important, but assistance in these areas was not as frequently provided as others they 
saw as less important. 

• Most of the surveyed students reported being satisfied with the assistance they 
received, with almost all supports from both the home and host institution having a 
mean rating of 4.0 or higher (on a scale of 1 to 5, where 1 means 
“completely unsatisfied” and 5 means “very satisfied”). 
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Received 
support 
% Yes 


Important 
% Yes 


Satisfaction 

Mean 

(5=very 

satisfied) 




N 


Obtaining financiai aid 


95 


84% 


96% 


4.3 


Academic recognition of credits or degrees 
earned 


93 


82% 


98% 


4.3 


Getting information on host institution 


90 


80% 


91% 


3.8 


Making academic contacts in host institution 


87 


77% 


89% 


4.0 


Heip with work piacement or internship 
(n=30)* 


24 


80% 


92% 


4.2 


Registration and course seiection 


74 


66% 


89% 


4.0 


Making social contacts in host institution 


64 


57% 


75% 


3.6 


Getting information on host country 


62 


55% 


81% 


3.7 


Arranging living accommodations 


57 


50% 


98% 


4.2 


Obtaining a visa or other immigration issues 


57 


50% 


88% 


4.2 


Obtaining health insurance 


34 


30% 


74% 


4.0 


Source: Survey of participating students 

Note: "Don't know," "no response," and "not applicable” categories are not Included. 
* Only students In an Internship or work placement answered this question. 



The two areas where surveyed students were less likely to receive assistance from their 
home or host institution were: 

• obtaining health insurance (Table 7 shows that 56 percent did not receive support from 
home institution; 50 percent received no support from host institution); 

• obtaining a visa or other immigration issues (Table 7 shows that 29 percent received 
no support from home institution; 27 percent received no support from host institution). 







N 


No support given 
% Yes 


Desire support 
% Yes 


Obtaining health insurance 


63 


56% 


48% 


Arranging living accommodations 


48 


43% 


65% 


Making social contacts in host institution 


42 


37% 


50% 


Getting information on host country 


40 


35% 


38% 


Obtaining a visa or other immigration issues 


33 


29% 


61% 


Help with work placement or internship 
(n=30)* 


6 


20% 


67% 


Registration and course selection 


20 


18% 


55% 


Making academic contacts in host institution 


19 


17% 


58% 


Getting information on host institution 


16 


14% 


81% 


Obtaining financial aid 


11 


10% 


64% 


Academic recognition of credits or degrees 
earned 


6 


5% 


83% 


Source: Survey of participating students 

Note: "Don't know," "no response," and "not applicable” categories are not included. 
* Only students in an internship or work placement answered this question. 
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Almost two-thirds of the surveyed students who took language preparation 
courses believe that the courses met their needs, while 29 percent believe that the 
courses did not meet their language needs. 

Almost half of the surveyed faculty believe that sufficient language preparation 
was offered, while a quarter believe that the language training was insufficient. 



The current provision of language preparation in I AM projects is not clear from the 
survey data. This may be due to the small samples, particularly in the student survey. 
Table 8 compares the student and faculty survey results. 











Faculty 

(n=40) 


Students 

(n=113) 


Didn’t prepare; host country used student’s first ianguage 


13% 


35% 


Didn’t prepare; student was fluent in the country’s language 


23% 


22% 


Student took preparation courses for credit 


30% 


12% 


Student took preparation courses, but not for credit 


40% 


15% 


Student wasn’t fluent but didn’t take language course 


- 


15% 


No language preparation courses were provided 


25% 


- 


Source: Survey of participating students and survey of project directors and educational partners 
Note: "Don't know" and "no response” answers are not included. 



The surveyed students reported an average length of 21 weeks for the language 
preparation courses. The median was 11 weeks and the range was from less than one 
week (3 percent or n=l) to 104 weeks (3 percent or n=l). The student survey also 
indicates that the lAM projects either fully funded language preparation or did not cover 
it at all. 

Almost two-thirds (65 percent or n=20 of 31) of the surveyed students who took language 
preparation courses believe that the course met their needs for the exchange project. 
Twenty-nine percent (n=9) thought that their language needs were not met by the course, 
and 3 percent (n=l) thought the preparation exceeded what they needed for the exchange. 
Of the students who were not fluent in the host country’s language (either as a second or 
first language) and went on their exchange without any language preparation (n=17), 
6 percent (n=l) believed that the lack of preparation caused difficulties during the 
lAM project. 

Almost half of the surveyed faculty members (48 percent or n=19) believe that students 
receive sufficient language preparation for their study abroad, compared to a quarter 
(23 percent or n=9) who find the language preparation to be insufficient.^^ To improve 
language preparation, two (5 percent) of the surveyed faculty suggested providing 
funding for language instruction, and two (5 percent) others proposed selecting students 
sooner to provide more time for language instruction. 



Some of the categories were not asked in both surveys. 

The remaining faculty members either did not respond (18 percent) to the question, or their partner institutions used 
the students’ first language (13 percent). 
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According to the surveyed students and faculty, the students experienced few 
problems with getting academic credit for their studies or work abroad. 



The evaluation conducted by the European Union of its Co-operation Programme in 
Higher Education and Training between the European Community and Canada 
(the EC-Canada Programme evaluation) concluded that it was important to plan students’ 
academic studies or their work/internship prior to their going abroad. Eikewise, other 
research examined by the literature review showed that ensuring credit recognition and 
transfer of credits earned abroad is important to the success of academic mobility 
programs. Previous studies indicated that students might not participate if they were 
uncertain about receiving academic credit for their work abroad. 

The surveys of participating students and faculty members appear to show that the 
lAM projects in both programs provided advance planning for their students’ exchanges. 
Over four-fifths of the students reported that they planned their academic studies 
(87 percent or n=93) or internship (87 percent or n=26) at least partly before going 
abroad. Similarly, two-thirds (68 percent or n=27) of the faculty members indicated that 
students’ academic study programs were decided upon before the students left their home 
institution, and one-fifth (20 percent or n=8) noted that students began this planning at 
the home institution, even if they completed it abroad. Of those faculty members who had 
students in internships (n=21), 76 percent (n=16) had students do at least some of the 
planning at their home institution before going abroad. 

Both surveys also appear to show that students do not experience much difficulty with 
getting academic credit for their studies or work done abroad. Eighty-seven percent 
(n=98) of the surveyed students reported receiving academic credit at their home 
institution for the courses or work they did in the lAM project. Eor 88 percent (n=86 of 
98) of them, the credit was equivalent to what they would have earned had the 
study /work been done at their home institution. Of the 15 students who did not receive 
academic credit for their work, 13 did not request academic credit. 

Similarly, almost two-thirds (68 percent or n=27) of the surveyed faculty members 
reported that their mobile students enrolled in courses that would earn them credit at their 
home institution. Of the faculty whose students received academic credit, over 
three-quarters (71 percent or n=25) reported that their students received equivalent credit 
for their courses abroad. 

Three faculty members (8 percent) reported that a minority of their students had difficulty 
receiving academic credit at their home institution for the study or work done abroad. 
They noted several reasons why the students did not receive credit. Most of their reasons 
were institutional and not the responsibility of the student, such as the structures of the 
national education systems being too different, the grading systems being too different, a 
lack of coordination between institutions, and the host institution failing to provide 
sufficient information on the program abroad. 



The percentages are based on the students who participated in each type of exchange, which explains why the 
percentage of internship students is the same as for academic studies students while the numbers of students are very 
different. 
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The project information collected by project directors and educational partners 
was most often used for informal evaluations of the project and for 
HRDC’s annual reports. 



Project directors and educational partners reported collecting various kinds of project 
information. Because of the high “don’t know” response rate, only the answers given by 
project directors, who most often handle the administrative work of the projects, are 
provided in Table 9. The results show that financial information and students' experiences 
are the records most often kept by project directors. 









Yes - Project directors 


Students' experiences 


21 


84% 


Financial information 


21 


84% 


Project-sponsored activities 


16 


64% 


Project’s impacts 


12 


48% 


Source: Survey of project directors and educational partners 







The more detailed analysis in the technical report shows that less than a majority of all 
the surveyed faculty members (project directors and educational partners) reported 
collecting information on the project’s impacts on the post-secondary institution, 
students, faculty, the department, etc. 

The information collected by project directors and educational partners is most often used 
for informal evaluations of the project and for HRDC’s annual reports. A third 
(33 percent or n=13 of 40) of the surveyed faculty members reported using the 
information in formal evaluations of the project. 



3.3 Short-Term Benefits 

The formative evaluation examined three evaluation issues regarding short-term benefits: 

• Has lAM encouraged international exchange and co-operation among Canadian higher 
education institutions (universities and colleges)? Specifically, in regard to: 

• setting up new multilateral partnerships between educational institutions and 
enhancing existing ones; 

• skills development and international academic work credit; 

• the development of common and/or shared curricula; and 

• the exchange of expertise and knowledge in new developments in higher education 
and training, including distance learning? 

• Has lAM improved the quality of human resource development among mobile and 
non-mobile students at participating institutions? Specifically, in regard to: 
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• increasing students’ knowledge of languages, cultures, and institutions; 

• providing affordable access to an “internationalized” curricula; 

• providing work placements/internships in a “foreign” milieu; and 

• partnerships beyond the educational field to include businesses, professional 
associations, public authorities, etc.? 

• Are the partnerships created through the lAM program continuing beyond the program 
funding? What mechanisms are being used to promote sustainability? 



Perceived Institutional Effects 



The surveyed faculty members rated lAM’s contribution to establishing 
partnerships and enhancing existing partnerships as important to their 
institution. However, faculty members wanted lAM to do more to enhance 
existing partnerships by allowing continued funding of successful lAM projects. 



When asked about the lAM Initiative’s contributions to their institution, the surveyed 
faculty members considered the most important contribution to be assistance in providing 
exchange experiences to students and assistance in establishing new partnerships or 
enhancing existing ones. Specific types of co-operation, such as forming partnerships 
with non-academic organizations, developing common curricula, and exchanging 
knowledge in new technologies scored lower in importance. 

As noted in section 3.2, faculty members wanted lAM to do more to enhance existing 
partnerships by allowing continued funding of successful lAM projects. 

The surveyed faculty members reported that most consortium collaboration was 
done informally, rather than by formal agreement. Formal collaboration occurred 
most often in the areas of academic credit recognition and transfer agreements, 
common curriculum materials, common courses or course modules, and faculty 
or staff exchanges. 

Eighty percent of the surveyed faculty members reported reaching agreements for 
credit recognition, and half considered the agreements reached on credits to be 
capable of serving future students. 



Most consortium collaboration consisted of informal agreements. Formal agreements 
were most common for credit recognition and transfer agreements, common curricular 
materials, common courses or course modules, and faculty or staff exchanges. These are 
also the areas that faculty members considered the most successful. 

Eighty percent (n=32) of the surveyed faculty members reported reaching agreements for 
credit recognition. Half (50 percent or n=16 of 32) considered the agreements reached on 
credits to be capable of serving future exchange students. Twelve respondents indicated 
that their agreements would only serve the current exchange project. 
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Seven faculty members reported using some form of the European Credit Transfer 
System (ECTS). The ECTS, developed for the European Community’s 
ERASMUS program for student mobility, is a voluntary, decentralized system of credit 
agreements that is based on institutional co-operation and trust. One advantage in 
adopting a system like the ECTS is that it sets in place a system that could potentially 
benefit other students. 

The surveyed students reported little difficulty in obtaining academic credit (usually 

equivalent) for their studies or work abroad. The figures appear comparable to the 
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EC-Canada Programme evaluation results. 

About two-thirds (68 percent) of the surveyed faculty members reported that 
collaboration with their partner institutions has continued after lAM funding 
ceased. The continued collaboration occurs mostly in the areas of student and 
faculty exchanges. 

Collaboration in common curricular materials and courses appears to decline 
substantially after the funding ends. 



About two-thirds (68 percent or n=27) of the surveyed faculty members reported that the 
collaboration with their partners has continued since their lAM funding ended. Table 10 
shows that almost one-quarter (23 percent or n=9) of the surveyed faculty members 
consider their collaborative activity to have increased; however, a slightly greater number 
(28 percent or n=ll) believe that their collaboration had declined. Three (8 percent) of 
the surveyed faculty members reported that the collaboration ceased after the 
lAM funding ended. 



Table 10 

Continuation of coiiaboration with partners (n=40) 




# 


% 


Yes, but collaboration has declined 


11 


28% 


Yes, collaboration has increased 


9 


23% 


Yes, collaboration has continued at the same level 


7 


18% 


No, collaboration has ceased 


3 


8% 


DK/NR 


9 


23% 


Source: Survey of project directors and educational partners 



The types of activities where collaboration has continued are mostly in the area of student 
and faculty exchanges (as shown in Table 11), which are also the areas where faculty 
members reported the most collaboration during the project. Another area that appears to 
be successful in continuing collaboration is distance learning, where six of the surveyed 
faculty members reported continued collaboration (which is about half of faculty who 
reported collaborating in this area in their projects). In contrast, the two other areas of 
common curricular materials and common courses appear to experience a substantial 
drop-off in collaboration after funding ends, despite the fact that they are areas of 
substantial co-operation during the projects. 



** Ninety-two percent (92 percent) of EC partners said that students received formal certification at their home 
institution for their study abroad in Canada. 
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Table 11 

Types of continued collaboration (n= 40) 




# 


% 


Student mobility 


18 


45% 


Faculty or staff exchanges 


11 


28% 


Joint faculty research 


9 


23% 


Distance learning courses 


6 


15% 


Common curricular materials 


5 


13% 


Common courses or course modules 


5 


13% 


Partnerships with business 


3 


8% 


Team teaching opportunities 


3 


8% 


Other 


3 


8% 


DK/NR 


13 


33% 


Source: Survey of project directors and educational partners 

Note: Respondents could choose more than one answer. Totals may sum to more than 100% 



Faculty members listed money as one of the primary factors that detracted from project 
sustainability. Twelve faculty members said the lack of funds hurts sustainability of 
projects. Other reasons cited were: “too much paperwork,” “personnel change, different 
goals among directors,” “differences in professional/academic standards between 
institutions,” and “will of the partners under the constraints of HRDC’s program.” 
Two faculty members noted factors that contributed to sustainability: “ongoing personal 
relationships, shared interests, trust,” and “collaboration among professors allowed for 
the elaboration of a project with an international profile.” 



Perceived Student Benefits 



Seventy-five percent of the surveyed students agreed with the statement that their 
lAM experience has helped them with other academic programs. 

Almost half of the surveyed students believed that their academic progress abroad 
was greater than it would have been at their home institution. 



Seventy-five percent (n=85) of surveyed students agreed with the statement that their 
lAM experience has helped them with other academic programs. 

The surveyed faculty members also commented on the effects of the exchange experience 
on students’ academic careers. On a scale of 1 to 5 (where 1 means “strongly disagree” 
and 5 means “strongly agree”), faculty rated academic effects the highest (as compared to 
professional or personal effects) with a mean rating of 4.6.*^ 

Surveyed students were also asked to compare their academic progress abroad with what 
they think it would have been at home. The results were as follows: 

• Almost half (46 percent or n=52) of surveyed students believed that their academic 
progress abroad was greater than it would have been at their home institution. 



Twenty-one percent of faculty surveyed either did not respond or indicated that they were uncertain about 
any effects. 
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• Forty percent (n=45) characterized their academic progress during the I AM project as 
“about the same” as it would have been at home. 

• Fourteen percent (n=16) of surveyed students thought their academic progress abroad 
was less than it would have been at home. 

These results seem less positive than the findings of the literature review. More than half 
of the students in two other studies (the Study Abroad Evaluation Program and the 
ERASMUS evaluation) rated their academic progress abroad as greater than it would 
have been at home. More than 80 percent of EC students in the EC-Canada Programme 
characterized their progress as greater. The small sample size of the student survey 
conducted for the lAM evaluation precludes inferences to the population of participating 
students; however, this may be an area for further study. 

The literature review also noted that studies differ on whether students experience 
graduation delays due to participation in student mobility programs. The student survey 
conducted for the lAM evaluation found that 21 percent (n=24) of the students reported 
that they had to prolong their time to graduation because of their participation in the 
lAM project. The remaining 79 percent (n=89) said that their participation in lAM did 
not lengthen their time to graduation. 

The majority of the surveyed students believe that their language skills improved 
and that they have increased knowledge of other cultures and improved 
knowledge of international issues. The surveyed students also reported that their 
exchange experiences helped them become better able to adapt to new situations 
and meet life’s challenges. 

These results appear to confirm the literature review findings that students 
believe that they benefit personally from the exchange experience. 



The studies examined by the literature review found that students in international 
exchanges reported an increase in their foreign language proficiency. Eor the 
lAM evaluation, surveyed students whose host country did not use their first language 
(n=73) were asked to assess their foreign language skills after they participated in the 
lAM project. A majority (84 percent or n=61) of these surveyed students believed that 
their foreign language skills improved during the exchange. Three respondents said that 
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their language skills would have improved just as much in ordinary classes. 

Also, the results of the student survey appear to align with the literature review findings 
that students tend to place more emphasis on the personal effects of their exchange 
experiences than on the academic or professional effects. Over 90 percent of surveyed 
students either “strongly agreed” or “agreed” that the exchange experience had the 
following personal effects: 
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Eight percent (n=9) of surveyed students answered “not applicable” to the question. 

For the studies discussed in the literature review, students placed a greater emphasis on personal effects by 
considering their most positive professional effects to be in personal characteristics, such as personal development, 
work style, and social skills. 
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• improved their knowledge of international issues (96 percent or n=108); 

• helped them better adapt to new situations (94 percent or n=106); 

• helped them be better prepared to meet life’s challenges (95 percent or n=107); 

• helped them better understand other cultures (93 percent or n=105); and 

• gave them contacts they will keep for years (92 percent or n=104). 

The surveyed faculty members also agreed with many of the above personal effects (on a 
scale of 1 to 5, where 5 means "strongly agree"): 

• students are more sensitive to other cultures (mean of 4.5); 

• students are more mature and self-confident (mean of 4.5); and 

• students are more flexible when encountering new situations (mean of 4.3). 

Most of the surveyed student participants now employed full-time reported that 
their joh is related to the studies or work they did in the lAM project, and that 
they are using skills learned in the lAM project in their jobs. 

Most of the surveyed students believe that their lAM experience enhanced their 
professional mobility, increased their interest in working abroad, increased their 
interest in international business, enhanced their employability, and affected their 
career choice. Faculty agreed that, after their lAM experiences, students are in a 
better position to conduct international business and they will probably apply 
their lAM experiences to their employment. 



The student survey sought information on possible professional effects of the 
lAM project. While most of those surveyed were still full-time students, the following 
results are from surveyed student participants now employed full-time (n=60): 

• Eighty-two percent (n=49) reported that their job is related to the studies or work they 
did in the lAM project; 

• Eighty-eight percent (n=53) believe that they are using skills learned in the 
lAM project in their jobs;^^ 

• Almost half (48 percent or n=29) reported that their job has an international aspect; 
and 



This percentage appears to be much higher than in other studies that consider this issue. For a discussion of those 
studies, please see the Literature Review Report. 

In the literature on student mobility, the evaluations of exchanges found that a considerable number of former study 
abroad students did not find jobs with an international focus. According to the literature, whether alumni/alumna of 
the study abroad programs found employment with an international focus appears to depend on two things: their 
academic field and the status of their host country (such as the size of its population and its economy and whether it 
had an internationally important language). 
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• Of the surveyed students who used a foreign language in their host country (n=36), 
over a third (36 percent or n=13) said that they use the language of their host country in 
their job. 

A majority of students “agreed” or “strongly agreed” that they learned job-specific skills that 
will enhance their employability and had experiences that affected their career choice. 

Perceived professional effects that received the highest level of agreement from surveyed 
students were: enhanced professional mobility (91 percent or n=103 “agreed” or 
“strongly agreed”) and increased interest in working abroad (88 percent or n=100 
“agreed” or “strongly agreed”). 

The surveyed faculty members also agreed that students experienced certain professional 
effects. On a scale of 1 to 5, (where 1 means "strongly disagree" and 5 means "strongly 
agree"), the faculty members rated the statements that after the exchange, students are in 
a better position to conduct international business and they will probably apply their 
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lAM experience to their employment with mean ratings of 4.3. 

Overall, more than two-thirds of the surveyed students were “very satisfied” with 
their exchange experience. Only one percent of the surveyed students were not 
satisfied, and none were “completely dissatisfied”. 



On a scale of 1 to 5 (where 1 means “completely dissatisfied” and 5 means “very satisfied”) 
students rated their overall experience in the exchange with an average rating of 4.7. Over 
two-thirds (68 percent or n=77) of surveyed students were “very satisfied” with their 
exchange experience. One percent (n=l) of surveyed students were not satisfied. None of the 
surveyed students reported being “completely dissatisfied” with their experience. 

Students gave a variety of reasons for their satisfaction with the exchange experience, but 
a few of the highlights are as follows: 

• Cultural and personal experiences received the most comments from students. Some 
examples of what students said include: 

• “The people that I’ve met showed me so many different things about life - more 
than academic, it is the way I look at people and life.” 

• “I learned a lot, I met a lot of new people, I have a different way of seeing life.” 

• “I think it was a worthwhile experience to learn and study in another culture. I think 
I learned a lot in making my way in another culture.” 

• “Fabulous opportunity to make friends and learn about culture.” 

• “Invaluable cultural experience.” 



Twenty-one percent of faculty surveyed either did not respond or indicated that they were uncertain about using 
their experiences in their employment, and almost one-third (3 1 percent) were either uncertain or did not respond to 
the statement about international business. 
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• Academic reasons for student satisfaction with the exchange included improved 
language skills (n=12), exposure to different academic approaches (n=10), and that the 
project was generally academically rewarding (n=9). 

• “It was a great opportunity to learn that the Canadian way isn’t the only way.” 

• “It was an eye-opener for me to see how other government services operated and 
how we in Canada could improve our government services.” 

• “It was certainly useful to learn about what other people were studying and their 
different perspectives.” 

• “It gave me the opportunity to learn about the issues that are relevant to my career 
choice in an international context.” 

• A few students (n=5) mentioned career or employment effects. 

• “It gave me better employment opportunities.” 

• “I got a job immediately after the project in Mexico.” 

Non-mobile students became involved in the projects in many ways, but primarily 
through taking courses developed with the partner institution. Non-mobile students 
benefited from the international dimension of the jointly developed courses. 



Fourteen faculty members (35 percent) had non-mobile students from their institution 
involved in the project. About one-third (36 percent or n=5 of 14) had to limit the 
number of non-mobile students that could participate, largely because the project had a 
pre-determined limit on the total number of students that could participate. As Table 12 
shows, non-mobile students became involved in the projects in many ways, but primarily 
through taking courses developed with the partner institution. 



Table 12 

Forms of non-mobile participation (n=14) 




# 


% 


Took course developed with partner institution 


10 


71% 


Undertook joint projects/activities with students at partner institutions 


7 


50% 


Undertook projects/other activities with exchange students from partner 
institution 


6 


43% 


Buddied with visiting students from partner institutions 


5 


36% 


Took a course/module given by visiting foreign faculty 


4 


29% 


DK/NR 


1 


7% 


Source: Survey of project directors and educational partners 

Note: Respondents could choose more than one answer. Totals may sum to more than 100%. 
Base: Faculty members who had non-mobile students in their proiect. 



However, no non-mobile students participated in the student survey. 
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4. Methodological Issues for the 
Summative Evaluations 



This section discusses a range of methodological issues for future evaluations of the 
lAM Initiative. The first part of the section highlights the following themes: 

• creating a valid comparison group for the lAM is unlikely; 

• outcomes must be more clearly defined; 

• attribution is unlikely; and 

• logistics must be managed. 

With these themes as a backdrop, the latter part of this section outlines a potential design 
for summative evaluations of the lAM programs. 

4.1 Creating a Valid Comparison Group for the lAM 
is Unlikely 

In evaluation, as in science, comparing outcomes for those who receive a program with 
those who do not is the accepted method for inferring incremental impacts of a program. 
However, it is very unlikely that a valid comparison group could be obtained for 
summative evaluations of the lAM Initiative. The approaches reviewed below are either 
not possible in the case of lAM or would require costly coordination and the participation 
of many institutions. 



Randomization 

The ideal method for creating a valid comparison is randomization. By randomly 
assigning subjects to a program or treatment group, it is possible to ensure that there are 
no statistically significant differences between the treatment group and the control group 
of non-participants. This method assumes that the sample sizes of the two groups are 
large enough and are drawn from a large population. Any observed differences can then 
be attributed to program interventions. Randomization is rarely possible in policy settings 
or socio-economic program settings, however, because ethical and political concerns 
require that programs be available to all. 

The central problem with using actual participants and non-participants, instead of 
randomly assigned subjects, is that those who apply and are accepted into programs 
(participants) may differ from those who do not apply or are not accepted 
(non-participants). For example, participating students may have superior grades or more 
desire to travel abroad. Therefore, observed differences between lAM participants and 
non-participants could be due to pre-program differences between these two groups. 
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Using Secondary Data Sources 

Another approach is to consider whether it is possible to create a useful comparison 
exercise between program participants and non-participants using secondary data sources. 
For example, if program participants and non-participants have drivers’ licenses, 
consistent measures such as age and gender may be obtained for both the participant and 
non-participant groups. Health insurance databases and school records could also be used 
to generate information on the two groups. These “background” variables can then be 
used in multivariate analysis to control for differences between the two groups, with a 
“dummy” variable inserted into the statistical model to capture any effects arising from 
the program. However, there are several limitations to this approach: 

• A “complete” set of background variables must be collected to control for all possible 
impacts on outcomes caused by individual attributes. Few databases offer a complete 
set of data, requiring researchers to try to assemble information from several sources. 

• It is difficult to know what constitutes a complete set of background variables, and one 
always wonders whether a crucial variable has been omitted that could account for the 
observed differences in outcomes. 

• Increasing the number of control variables to a multivariate specification produces a 
range of statistical difficulties.^^ 



“Matching” Participants to Non-Participants 

Yet another approach to the construction of a comparison group is to “match” 
participants to non-participants based on attributes believed to determine the outcomes. 
For example, a participant who is single, a student, aged 19, female, in engineering, and 
from Alberta would have a comparison group “match” who has all these same attributes. 
Clearly, matching on a wider range of variables increases the power of the comparison 
process, but raises the data demands because exact matching requires the same 
information on both participants and non-participants. A less exact approach is to match 
on ranges, such as an age range of 18 to 21, rather than on the exact age of 19. Taking the 
match-on-ranges approach lessens the data demands but reduces the power of the test. 



These include multicollinearity and simultaneous equation bias. The latter occurs when a variable on the right hand 
of the regression model is also determined by the dependent (left hand variable). Both these problems reduce the 
precision of the statistical estimation and invalidate the model. 
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Using a Group Norm Approach 

Finally, one can compare the average experience of program participants with a group 
norm by using, for example, the National Graduates Survey (NGS). For this method, it is 
necessary to select a relevant outcome variable for the comparison. Of course, this 
approach completely ignores the problem of self-selection. To increase the precision of 
the test, HRDC could arrange with Statistics Canada to extract a sub-sample from the 
NGS that more closely resembles the lAM participants than the total sample, but this 
would be time-consuming. Another approach is a national survey of universities that 
surveys undergraduate students. It may be possible to include several questions on this 
survey to provide a reference point for the lAM. 



Implications for I AM Evaluations 

All of the above approaches are either not possible or would require costly co-ordination 
and the participation of many institutions. For example, creating a comparison group 
from secondary data sources for the lAM will require co-operation from the participating 
schools. Students typically provide a wealth of family and background data on the 
applications for admission and financial assistance. As an lAM project is approved, 
HRDC would need to arrange with school administrations to secure information that 
would support the identification of a comparison group. Registrars and college or 
university records would need to be approached to identify non-participating students to 
include in the comparison group. Further, all students (participants and non-participants) 
would have to consent to the collection of their personal data and would need to be 
offered incentives to encourage their inclusion in follow-up surveys. 

Also, arrangements for comparison groups would need to be made with each 
participating school and that would be too expensive to be feasible. Furthermore, few 
universities and colleges would be willing to share student information to create an 
effective comparison group. 

Other reference groups, such as the National Graduates Survey pose few questions 
relevant for testing the goals of the lAM. 



4.2 Outcome Variables Must be More Clearly Defined 

The lAM Initiative has intended outcomes for institutions and students. In the case of 
institutions, project directors and educational partners could report on outcomes for their 
institutions and their own careers in follow-up surveys that occur within a year or two of 
the project. By then, most of the institutional and professional outcomes should be 
apparent to participating faculty. 

The evaluation of student outcomes is more problematic, however. Student outcomes, 
such as career and other gains, will be difficult to detect because of the problems 
involved in creating valid comparison groups and the complexity of defining sensible. 
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long-term outcome measures. Most university/college graduates will find employment 
and, on average, will experience career progress in the form of higher wages and 
increased responsibility. The question is whether lAM participants show a statistically 
significant advantage in these areas. Therefore, establishing a norm from the comparison 
group is essential. If one confines outcomes to income, however, the net differences may 
be small. Selecting an outcome that reflects a goal of the lAM Initiative, such as success 
in pursuing a career related to international trade or other "global" careers, requires that 
non-participants who select these careers are also identified and monitored. Identifying 
these non-participants while they are at college or university will be challenging. 

Also, most students participate in lAM projects during the third or fourth year of their 
post- secondary education. Some go on to pursue graduate or professional training, and 
this training can often last for five years or more after they have completed their 
undergraduate degrees. Although, their experience with the lAM Initiative may affect 
their choice of post-graduate training and eventually may pay dividends in their careers, 
these sorts of effects can occur many years after program participation. 



4.3 Attribution is Unlikely 

The lAM Initiative, like most international exchange programs for post- secondary 
students, is intended to increase the chance that a student will become engaged in a career 
demanding awareness and experience with foreign cultures. However, it is unlikely that 
initiatives such as lAM will have a measurable impact until a student has been in the 
workforce for several years. It is also likely that a wide range of other factors influence 
career choice and progress, while the influence of a student’s lAM experience can be 
considered modest or finite and of declining importance as an individual matures and 
works up the career ladder. 

Baseline data and comparing participants to the "average" student at an institution is the 
route to successfully controlling selection bias. In turn, this means that the program may 
wish to consider collecting detailed information on students at the outset of 
their participation. 



4.4 Logistics Must be Managed 

Securing survey responses from students and faculty members will always present major 
challenges. Students are highly mobile and, for a large fraction of participants, address or 
phone information will become out of date as early as three months after the project. 
Because students participating in lAM projects are usually in their third or fourth year, 
few will have the same address and phone number after two years. This has 
three implications for the research: 

• Follow-up surveys need to commence very soon after program completion. 
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• Collateral information on the student's permanent address, family contact or a friend 
who will know his/her whereabouts is critical because, without it, any long-term 
follow-up will surely fail. 

• Most importantly, the address and collateral location information might be collected at 
the time of participant application for all students and faculty. This will improve the 
possibility of conducting follow-up surveys. 

• Also, it is preferable to collect at the time of student participant application. Failure to 
secure consent will require a convoluted process that involves an initial mailing and 
then awaiting the consent card. Experience here shows the futility of that approach. 

Project directors present some different challenges. Several refused outright to participate 
in the formative evaluation, citing the administrative burdens of previous reports and the 
fact that they had completed a lengthy questionnaire associated with an audit of the 
program. The length of the questionnaire discouraged some from participating. 

The academic cycle produces optimum windows for academic surveys April and 
December at exam time are best. October to November, January to March, and May are 
reasonable times for surveys of faculty. Summer is very poor. 



4.5 Suggestions for the Summative Evaluation 

With these four themes as background, it is possible to outline a methodology for the 
summative evaluation of the lAM Initiative. Given the problems associated with 
comparison groups and the difficulty in inferring attribution, the summative evaluation 
may wish to focus on collecting information on short-term outcomes and how the project 
has assisted the post-secondary institution to increase its capacity to participate in 
international exchange programs. 

Also, project applications, student applications, and project reports would be better used 
to anchor the evaluation. This approach would reduce the level of effort needed for 
follow-up surveys by making better use of the application process and by collecting key 
information from Project Directors in follow-up project reports. For example, the survey 
of Project Directors and Educational Partners can be eliminated entirely. 

The proposed lines of evidence for the summative evaluations are discussed below. 



Administrative Data 

Administrative data consists of the following data collection and analysis activities: 

• Project application (submitted by Project Director); 

• Student/participant application; 

• Project report submitted by Project Director at conclusion of the project; and 
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• Program database development and annual reporting. 

The project application could provide information about the proposed project as well as 
the participating faculty. It is important to collect attributes of the Project Director and 
Educational Partners such as: 

• Other research grants (general and those associated with foreign research); 

• Number of graduate students supervised; 

• Academic rank, years of employment at present and all academic institutions; and 

• Highest degree and year of graduation. 

The standard applications for all students could be designed to collect the following 
information: 

• Name, current address and phone; 

• Secondary contact information (name address of parent or other who will know where 
the student will be); 

• Current academic program, grade point; 

• Work history (last 3 positions); 

• Experience with international travel; 

• Capacity in additional language; 

• Expectations for the program; 

• Personal and family socio-economic status (occupation of parents, family 
income, etc.); and 

• Informed consent release. 

The onus for ensuring that all student and educational partner applications are complete 
could be placed on the Project Directors. In effect, the application function will provide 
partial information normally collected by follow-up surveys. 

Reports by Project Directors submitted at the conclusion of the project could supply a 
range of information on activities and short-term outcomes from the perspective of the 
participating faculty members. This includes number of students, duration of stay, total 
costs incurred in the exchange, all sources of funding not covered by the lAM 
(i.e., cross-supported by other grant funds), activities in the project, etc. Project Directors 
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could collect input from educational partners using a standard questionnaire.^’ This will 
ensure that the experience of all educational institutions is included in the report. 

The lAM program could develop a standard reporting template with an instruction guide 
to increase uniformity of response. Also, the annual report could include feedback from 
Educational Partners on the contribution made by the lAM project for their students and 
institutions. The Project Director can submit all feedback using the same template. 

A program database could be developed using the project and student applications and 
Project Directors' report. This needs to be a proper relational database that can support 
queries and annual reporting for the program. 

The proposed approach relies both on the creation of a standard project report template 
and on Project Directors supplying additional information on the Educational Partners at 
the conclusion of the project. This raises the burden on Project Directors during the 
project, but eliminates the need to participate in a follow-up survey. lAM could consider 
allowing a small portion of project funds (i.e. $2000) to be used to assist with this 
administrative burden. 



Surveys of Students 

lAM may want to consider using application forms to eliminate the need for an entry 
survey. The follow-up survey of students could then be configured in three waves: 

First Wave; Students would complete an exit survey within a month of their return or 
completion of the program (and before classes end in April). The questionnaire would 
collect much of the information included in the survey used in this formative evaluation. 

This survey must request confirming details on the secondary contact (permanent 
address) for follow-up surveys. The questionnaire could contain a page (release) that the 
student signs indicating their agreement for HRDC to solicit student co-ordinates from 
the secondary contact. 

Second Wave: A second follow-up survey would be completed a year after students 
return. HRDC would initiate the process by sending a letter to the secondary contact 
(with a copy of the release signed by the student) requesting the current phone number 
for the student be provided (to a toll free line). A follow-up call may be needed to collect 
this information. The student survey would then proceed using a telephone interview and 
would collect information of current educational and labour market activity. 

Third Wave: Subsequent surveys are possible, with periodic re-contacting of the 
secondary contact, although attrition will likely increase over time. 

The survey data can be added to the program database to support an integrated 
information system for the summative evaluation. Participants may be cross-referenced to 



In general, more information value exists in collecting information from Project Directors than Educational Partners. 
Project Directors usually have the most information and typically will have more contact with students and the 
institutions in other countries. 
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the project and vice versa to support analysis of outcomes by project type, and even 
Project Director attribute and institution. Changing the follow-up questions on the 
surveys will impede longitudinal analysis and will use the program database to support 
the summative evaluation. 

The formative evaluation also provided a few important lessons learned in the execution 
of student surveys. For example, prize draws are a useful incentive to encourage students 
to participate in the research. Questionnaires must be concise and brief. All the 
questionnaires used in the evaluation were too long and attempted to collect much 
“nice to have” as opposed to “need to have” information. Also, if the academic cycle is 
not taken into account when scheduling the surveys, the response rates will be very poor. 
This is important for all phases of data collection for the summative evaluation. 



Qualitative Data 

Key Informant Interviews: Key informant interviews with program managers, 
university/college representatives (such as Association of Universities and Colleges of 
Canada and Association of Canadian Community Colleges), and senior management 
within HRDC provide important program details and context. As an option, key 
informant interviews may also be conducted with selected Project Directors, but it is 
likely that few will be able to offer additional information beyond that provided in the 
project report. 

Case Studies and Focus groups: Case studies and focus groups with students can offer 
useful insights into program operation. Case studies would also be useful for obtaining 
greater insight into projects with non-mobile students. For example, interviews with 
Project Directors, focus groups with participating students, and a review of curricula 
would constitute a useful case study. 

Case studies and focus groups must be conducted within a narrow time frame 
(i.e. immediately after students have returned). In particular, focus groups must be 
convened before April 1 because students disperse immediately after classes have 
concluded. Missing the narrow window for scheduling the focus groups with students 
would, however, erode this as a line of evidence. 
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5. Conclusion 



Small sample sizes limit any inference from the survey results to the larger population of 
lAM participants or unsuccessful applicants. Some tentative conclusions are possible, 
however, based on the literature review and the information gathered by the 
formative evaluation. 

• The concept of international student exchange is widely supported within the academic 
community. Both faculty and students endorse the concept as do studies of 
international exchange in other countries. 

• It is difficult to identify outcomes in terms of changes to career choice or career 
progress, but most students believe the experience will be beneficial. 

• Consideration could be given to allowing faculty to apply more than once. The 
program will benefit from having experienced faculty run projects. To ensure access to 
a wider number of faculty members, the program could limit the number of awards 
within any five-year period. 

• Unsuccessful project applicants require additional feedback. These applicants should 
be viewed as a resource, that with some coaching, could become proponents of 
successful projects. 
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